ci: add preview smoke test for frontend PRs by talissoncosta · Pull Request #7192 · Flagsmith/flagsmith

talissoncosta · 2026-04-09T13:10:24Z

I have read the Contributing Guide.
I have filled in the "Changes" section below.
I have filled in the "How did you test this code" section below.

Changes

Contributes to #7262

Vercel Preview Smoke Test

Adds a smoke test job to the frontend PR workflow. On every PR that touches frontend/**, the job waits for Vercel to finish deploying, then verifies the preview is healthy.

How it works

Phase 1 — Wait for Vercel deployments:

Polls the commit status API for flagsmith-frontend checks (scoped to frontend only — won't wait for docs)
Waits until all leave pending (real builds: waits for completion; skipped builds: immediately done)
If a build fails, reports the failure and skips testing

Phase 2 — Smoke test the preview:

Parses flagsmith-frontend-* URLs from the Vercel bot PR comment (short poll with 30s buffer for comment update timing)
GET /config/project-overrides — Express server is running
GET / — static files deployed + response body contains "Flagsmith" (content validation)

Properties

No false positives — Phase 1 ensures the current build has landed before testing
Non-blocking — continue-on-error: true, failures warn but don't gate merge
Bounded — timeout-minutes: 15 on the job, --max-time 30 on each curl
Handles skipped builds — Vercel's "Ignored Build Step" completes immediately; previous preview URL is still live
No checkout needed — uses only gh api + curl

Review feedback addressed

Job-level timeout
Curl timeouts (--max-time 30)
Non-blocking (continue-on-error: true)
Merged duplicate API calls
Dropped /health (behind VPN, returns 404 on public URLs)
Body content validation on /

How did you test this code?

Validated on a throwaway test PR (test: Validate smoke test (two-phase approach) #7264) with a real Vercel deployment
Previous iteration caught a false-positive issue — smoke test passed before Vercel finished deploying. Fixed with the two-phase approach.

🤖 Generated with Claude Code

vercel · 2026-04-09T13:10:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Apr 16, 2026 2:08pm
flagsmith-frontend-preview	Ignored	Preview	Apr 16, 2026 2:08pm
flagsmith-frontend-staging	Ignored	Preview	Apr 16, 2026 2:08pm

codecov · 2026-04-09T13:21:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.29%. Comparing base (9943da9) to head (27cf50f).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7192   +/-   ##
=======================================
  Coverage   98.29%   98.29%           
=======================================
  Files        1351     1351           
  Lines       50752    50752           
=======================================
  Hits        49885    49885           
  Misses        867      867

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Failures now warn instead of hard-failing individual endpoints. The job still exits non-zero so the PR check shows a warning, but continue-on-error prevents it from blocking merge. Also consolidates the 3 separate curl checks into a single endpoint loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The state and environment_url were fetched in 2 separate calls to the same endpoint. Now parsed from a single response — halves the API calls per poll iteration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

/health, /admin, /metrics and /sales-dashboard are now only reachable via internal.flagsmith.com (VPN). Public preview URLs return 404 for these paths. Keep /config/project-overrides (Express server-side route, proves the server is running) and / (proves static files are deployed). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

talissoncosta · 2026-04-16T13:03:05Z

Update: the previous approach produced false positives — it was testing URLs before Vercel finished deploying, hitting the stale previous build.

Rewrote to a two-phase approach:

Wait for Vercel deployment statuses to complete (polls commit status API for flagsmith-frontend checks)
Parse URLs from the bot comment and test /config/project-overrides + / with content validation

Also dropped /health (behind VPN now). Validating on #7264.

The Deployments API approach failed for "Ignored Build Step" skips — Vercel creates no deployment for the SHA, and stores SHAs as the ref field so branch-name queries also returned nothing. Parse the preview URLs directly from the Vercel bot PR comment, which is always present and has working URLs regardless of whether a build ran. Added pull-requests:read permission for comment access. Dropped /health from the checked endpoints — now behind VPN (internal.flagsmith.com only) and returns 404 on public URLs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Grep pattern widened from `flagsmith-frontend[^ )]*` to `[^ )]*vercel\.app` — resilient to project renames - Index page check now validates the response body contains 'Flagsmith', catching broken deployments that serve error pages with 200 status. Single curl call for both status + body. Closes #7262 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Previous approach tested preview URLs immediately, producing false positives when Vercel was still deploying (the stale previous build responded 200). Two-phase approach: 1. Wait for Vercel deployment statuses to leave 'pending' — polls the commit status API until all Vercel-related checks complete. Handles real builds (waits for success/failure), skipped builds (immediately complete), and build failures (reports + exits). 2. Parse frontend preview URLs from the Vercel bot PR comment (now guaranteed up-to-date) and curl them with status + content validation. Trade-off: slower (waits for Vercel build, typically 2-8 min) but correct — no false positives. Contributes to #7262 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Filter statuses by 'flagsmith-frontend' instead of 'Vercel' so docs deployments don't block the wait - Poll up to 30s for the Vercel bot comment after deployments complete (handles timing gap between deploy finish and comment update) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions bot added ci-cd Build, test and deployment related front-end Issue related to the React Front End Dashboard and removed ci-cd Build, test and deployment related labels Apr 9, 2026

vercel bot deployed to Preview – flagsmith-frontend-staging April 9, 2026 13:12 View deployment

vercel bot deployed to Preview – flagsmith-frontend-preview April 9, 2026 13:12 View deployment

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 9, 2026

vercel bot deployed to Preview – flagsmith-frontend-staging April 9, 2026 13:23 View deployment

vercel bot deployed to Preview – flagsmith-frontend-preview April 9, 2026 13:23 View deployment

talissoncosta had a problem deploying to staging April 9, 2026 14:14 — with GitHub Actions Failure

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 9, 2026

vercel bot deployed to Preview – flagsmith-frontend-staging April 9, 2026 14:17 View deployment

vercel bot deployed to Preview – flagsmith-frontend-preview April 9, 2026 14:18 View deployment

talissoncosta had a problem deploying to staging April 9, 2026 14:20 — with GitHub Actions Failure

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 9, 2026

vercel bot deployed to Preview – flagsmith-frontend-preview April 9, 2026 14:24 View deployment

vercel bot deployed to Preview – flagsmith-frontend-staging April 9, 2026 14:24 View deployment

talissoncosta had a problem deploying to staging April 9, 2026 14:46 — with GitHub Actions Failure

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 9, 2026

github-advanced-security AI found potential problems Apr 9, 2026

View reviewed changes

Comment thread .github/workflows/frontend-deploy-production.yml Fixed

Comment thread .github/workflows/frontend-deploy-production.yml Fixed

talissoncosta changed the title ~~ci: Add Vercel preview smoke test for frontend PRs~~ ci: Stage Vercel deployment and run E2E before promoting to production Apr 9, 2026

github-actions bot removed the ci-cd Build, test and deployment related label Apr 9, 2026

talissoncosta force-pushed the ci/frontend-smoke-test-on-vercel-preview branch from 89b5264 to 0c057e9 Compare April 16, 2026 12:30

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 16, 2026

talissoncosta and others added 4 commits April 16, 2026 09:32

ci: Add job-level timeout to smoke test (10 min)

9a4f381

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ci: Add --max-time 30 to smoke test curls

463a059

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ci: Merge deployment status API calls into one

06cd573

The state and environment_url were fetched in 2 separate calls to the same endpoint. Now parsed from a single response — halves the API calls per poll iteration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

talissoncosta force-pushed the ci/frontend-smoke-test-on-vercel-preview branch from 0c057e9 to 06cd573 Compare April 16, 2026 12:33

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 16, 2026

talissoncosta force-pushed the ci/frontend-smoke-test-on-vercel-preview branch from 0ab5559 to c7492fd Compare April 16, 2026 12:43

github-actions bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Apr 16, 2026

This was referenced Apr 16, 2026

CI: Known limitations of the Vercel preview smoke test #7261

Closed

CI: Add Vercel preview smoke test to frontend PR workflow #7262

Open

talissoncosta and others added 3 commits April 16, 2026 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add preview smoke test for frontend PRs#7192

ci: add preview smoke test for frontend PRs#7192
talissoncosta wants to merge 14 commits intomainfrom
ci/frontend-smoke-test-on-vercel-preview

talissoncosta commented Apr 9, 2026 •

edited

Loading

Uh oh!

vercel bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

codecov bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

talissoncosta commented Apr 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

talissoncosta commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Vercel Preview Smoke Test

How it works

Properties

Review feedback addressed

How did you test this code?

Uh oh!

vercel bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

talissoncosta commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

talissoncosta commented Apr 9, 2026 •

edited

Loading

vercel bot commented Apr 9, 2026 •

edited

Loading

codecov bot commented Apr 9, 2026 •

edited

Loading

talissoncosta commented Apr 16, 2026 •

edited

Loading